Replication of CORBA Objects

ions. The OMG has published guidelines for designing object services and their interfaces [OMG97]. The service approach has also been adopted in the Distributed Object-Oriented Reliable Service (DOORS) [CHY+98], currently in development at Bell Labs. Our work on CORBA fault-tolerant services has signi cantly in uenced many of the proposals that have recently been made to the OMG in the context of its undergoing standardization e ort towards fault-tolerant CORBA [OMG98b]. 4 The Object Group Service 4.1 Overview The Object Group Service (OGS) is a CORBA service that provides group management and communication facilities in a CORBA environment. It is composed of a set of generic IDL-speci ed interfaces. With OGS, clients can send invocations to object groups without knowing the number and identities of group members. In addition, OGS provides support for transparent group invocations, allowing clients to invoke operations on object groups as if they were invoking singleton objects. OGS is based only on standard CORBA mechanisms and is thus portable to any compliant ORB implementation. OGS may be used from any programming language that is supported by CORBA, or from any system that supports the CORBA Internet Inter-Orb Protocol (IIOP). More precisely, the OGS environment speci es an architecture and a set of IDL interfaces for object group support, as well as a set of object services that provide various facilities for reliable distributed computing. OGS uses a component-oriented approach. Basic units of functionality are packaged as CORBA services, which represent distributed components. These services, which include distributed agreement protocols and detection of remote component failure, are used in the implementation of the group communication primitives (see Sect. 4.3). Note that the relationship between these di erent services are implementation dependencies, and are not expressed in the IDL interfaces. 4.2 Transparency A common goal of object-oriented middleware environments is to hide the lowlevel mechanisms used for remote invocations and object management as much as possible, to let the developer focus on the application-speci c problems. Similarly, when working with object groups, a desirable property is to hide the complex mechanisms used for group communication. This section describes the concept of transparency in the context of OGS and the underlying mechanisms used to provide it. Transparency in OGS. Transparency is the property of a system to be invisible, i.e., the degree to which application programs are unaware of the system. Transparency appears at di erent levels in the OGS architecture [GFGM98] and may be classi ed as follows: { Plurality transparency: the application objects have the illusion that they deal with singleton objects, although they interact with object groups. { Behavior transparency: the application objects are not aware of the replication policy and the protocols run by OGS in order to make the system behave consistently. { Type transparency: requests are performed via direct method invocations on the server's interface, rather than by explicit calls to a group communication API (e.g., via a multicast() operation), and explicit packing and unpacking of operation parameters into and from messages. Type transparency means that the application does not need to perform type conversions. Full transparency is achieved if the application program is completely unaware of the group service. OGS provides full client-side transparency, without even requiring the client code to be recompiled [Fel98]. The Bene ts of Transparency. Transparency provides a number of facilities which make it useful, if not necessary, to develop distributed applications with object groups. Some of the bene ts of transparency are outlined below. { Ease of use. Applications do not need to be written with replication in mind. { Less error prone. Application programmers do not need to write error prone code for inserting and extracting data into and from requests. { Reuse of existing code. Existing client code can be reused without modi cation. Server code requires only minor modi cations. Objects from existing frameworks can be made groupable by using multiple inheritance of interfaces and implementations. { Encapsulate behavior in the group. OGS knows about the operations invoked on object groups, and can associate di erent semantics to distinct requests on behalf of the object group. The protocols may vary depending on the invoked operation without the client's knowledge. The Limitations of Transparency. Client transparency is generally considered as a \good thing" for object replication. However, server transparency has several drawbacks for complex systems. When dealing with replica con guration, failure detection mechanisms, or advanced synchronization between replicas, explicit group management support is required. In some cases, transparent replication can make systems su er reduced performance due to the lack of control over how replication is performed. 4.3 OGS Components OGS implements group communication using a set of CORBA services. These services are not organized in a layered architecture, but as a set of orthogonal components with usage relationships between each other (component-oriented approach). Figure 5 presents an abstract view of the major components dened in the OGS architecture. Although this is not clearly visible on the gure, each component is speci ed independently and they all interact with each other through the ORB. The application may use any of these components directly. These components are: Join/leave groups Remove failed members Notify upon view change Reach agreement between a set of objects Distributed Consensus Group Multicast Unreliable group multicast Reliable group multicast Total order group multicast • • • • • • • Group Membership • • • • Monitor remote objects Notify upon failure Reliable asynchronous communication FIFO channels Monitoring Messaging Object Request Broker Operating System and Network Services Application Objects O G S C o m p o n e n t s Fig. 5. Overview of the OGS architecture 1. A Messaging Service that provides non-blocking reliable point-to-point and multicast communication. 2. A Monitoring Service that monitors objects and provides failure detection mechanisms in CORBA. 3. A Consensus Service that allows a set of application objects to solve the so-called distributed consensus problem [CT96]. This service is used to implement group multicast and membership protocols. 4. A Group Service that provides both group multicast and group membership. The dependencies between these components are implementation-speci c, and do not appear in the IDL interfaces of the services. All these components are described in details in [Fel98]. In the following, we focus on the group service only. 4.4 The Group Service The group service is the core of the OGS environment. It is the service that actually provides object group support, and which the application programmer has to deal with. It implements two functionalities: { Group membership manages the life cycle of object groups. It maintains the updated list of all correct group members. It provides support for joining and leaving groups, view change noti cation, and state transfer. A group membership service is generally associated with a failure detection mechanism for detecting group member failures. { Group multicast provides support for sending multicast invocations to all the members of a group, with various reliability and ordering guarantees. The group multicast and group membership services interact closely. In particular, multicast operations are de ned on object groups rather than on sets of unrelated objects, thus involving group membership. Therefore, both services are contained in the set of interfaces forming the group service. Group Membership. The role of a group membership service is to manage memberships in a distributed system on behalf of the processes that compose it. OGS supports dynamic groups, i.e., the composition of the groups can change over time. New members can join an existing group, explicitly leave it, or may be implicitly removed from the group because of a failure. Objects that wish to join a group do so by contacting the membership service, which updates the list of group members. Once admitted to the group, an object may interact with other group members. Finally, if the object fails or leaves the group, the membership service will again update the list of group members. Dynamic group membership involves two kinds of protocols: { A view change protocol, which is run each time the composition of a group changes. It ensures that every correct member of the group receives a view change noti cation, indicating the new composition of the group as a list of group members with mutually consistent rankings. View changes are totally ordered with each other. { A state transfer protocol, which is an atomic operation occurring during view change, when a new member joins an existing group. It consists in obtaining the state from a current group member, and giving it to the new member. This protocol ensures that the state of all group members is kept consistent upon membership changes. The view change protocol can terminate only after a state transfer is completed. The group membership interfaces de ne how OGS and group members interact. They are essentially composed of two types of objects (Fig. 6): (1) servicespeci c4 group administrator objects that enable group members to change their status in the group (e.g., join and leave the group); and (2) application-speci c groupable objects that enable OGS to call back to the application for view change and state transfer protocols. The GroupAdministrator interface is implemented by the service, and is used as a black box by the application. A group administrator is assigned to a single group at creation time, but there may be several 4 We distinguish service-speci c objects, de ned and implemented by the service and accessed by the application through their IDL interface, from application-speci c objects, implemented by the application and supporting some interface de ned by the service. group administrators on the same host or in the same process. The Groupable interface must be implemented by application objects that want to be members of a group. Group Multicast. Group multicast provides primitives for sending invocations to groups instead of singleton objects. OGS provides a rich set of group multicast primitives, adapted to various types of applications. OGS implements groups as open structures, and allows non-member objects to issue multicast invocations to groups. Multicast primitives can be classi ed according to their degree of reliability and their ordering guarantees. In addition, OGS provides two communication models: untyped and typed invocations. Multicast Reliability. Reliable multicast is a key mechanism for developing replicated applications. In the context of group communication, it means that all correct group members deliver the same set of messages (agreement), that this set includes all messages multicast to the group by correct objects (validity), and that no spurious messages are ever delivered (integrity) [HT93]. OGS provides both unreliable and reliable multicast primitives. The unreliable primitives may be useful for read-only operations, but should be used with caution, since it may corrupt the state of the group if all messages are not received by every member. Reliable multicast, in itself, does not ensure that group consistency is preserved; it is generally combined with an ordering guarantee. Multicast Ordering. In addition to reliability, message ordering is an important concern when dealing with one-to-many communication. The state of an object generally depends on the order in which it receives requests, and some special multicast primitive must be used to maintain a consistent ordering of requests received by all group members. The current version of OGS provides unordered, FIFO, and totally ordered multicast primitives. Multicast Replies. Just as an invocation to a singleton object may return a value, a multicast invocation may return a set of values (one from each target object). The client may want to wait only for the rst reply, for several replies, or even for no replies. OGS provides the following options for the number of replies expected by the client: { All-replies multicast invocations wait for replies from all group members. { Majority-of-replies multicast invocations wait for a majority of replies from the group members. { One-reply multicast invocations wait for the rst reply from any group member. { Zero-reply (synchronous) multicast invocations wait for the operation to complete on one group member, but does not return any reply to the client. { One-way multicast invocations do not wait for any reply from the group members, and do not block the client's execution thread. With either majority-of-replies or all-replies multicast invocations, OGS might need to modify the expected number of replies if a server crashes while the invocation is processed. The zero-reply synchronous invocation style is useful for ow control to avoid that the object group becomes congested; it blocks the sender until the message is processed, allowing the receivers time to decongest. Untyped Interfaces and Explicit Invocations. The untyped invocation interface of OGS allows clients to send untyped messages to group members. An object that wishes to send a multicast to an object group must explicitly pack the data of the message into a value of type any, and pass it to OGS, which will perform the multicast. On the server side, OGS delivers this message by passing the any value to group members through their deliver() operation. This invocation interface is exible since the client can place any kind of data in the message, and can easily specify the semantics associated with the multicast invocation and the expected number of replies. The drawback of this model is that the application programmer must explicitly pack all parameters associated with the server's invocation in a message (marshaling), and extract these parameters on the server side (unmarshaling), which is a painful and error-prone task. In addition, this untyped invocation interface exposes group communication to the client application, while a common goal to most group-based systems is to hide groups from clients. Typed Interfaces and Transparent Invocations. The typed invocation interface of OGS provides group transparency to clients, which can issue invocations to object groups as if they were invoking singleton objects. The client directly invokes operations of the server's interface using static stubs, and OGS delivers multicasts by directly invoking the relevant operation of the server, using static skeletons. OGS transparently lters messages and returns a single reply to the client. Of course, typed communication does require that all servers implement the same IDL interface. Although less exible than the untyped invocation interface, this model is much easier to use. The application developer does not need to perform the marshaling and unmarshaling of the request (these operations are performed transparently by OGS), and can bene t from the type safety of CORBA's static invocation interface. The client cannot specify the semantics to associate with a multicast invocation but, as we will see in next section, this may be advantageously replaced by server-speci ed invocation semantics. Server-Speci ed Invocation Semantics. As mentioned in Sect. 4.2, OGS provides server-speci ed invocation semantics: the server may associate speci c semantics to each individual operation of its interface when using the typed version of OGS.5 For instance, if an operation does not change the state of the server (read-only operation), the server may decide to deliver the invocation without 5 By default, OGS uses a total order multicast for invocations with unspeci ed semantics. ensuring total order. Since the server implements the operations, it knows their properties and the client does not need to be aware of them. Furthermore, the server can optimize client requests based on their semantics; for instance, two update operations do not have to be totally ordered with each other if they are commutative (e.g., they modify disjoint parts of the server's state). This approach is an improvement over the traditional model where a client asks for the strongest ordering guarantees for a message when it is unsure of the exact semantics of the associated operation. 4.5 OGS Components and Interactions OGS combines support for group membership and group multicast in a single set of interfaces. Figure 6 presents a simpli ed high-level view of OGS components. OGS interfaces are classi ed according to three categories, associated with the di erent views of the service: Group Members Client Groupable GroupAccessor GroupAdministrator Groupable GroupAdministrator π Object Group Service T Has an interface of type T to... Distributed Protocol Appl. Obj. Appl. Obj. Appl. Obj. Srv. Obj. Srv. Obj. Srv. Obj. Fig. 6. OGS components overview 1. Client interfaces (GroupAccessor) allow clients to interact with object groups. 2. Member interfaces (GroupAdministrator) are a superset of client interfaces, and allow servers to manage the group's life cycle (e.g., join and leave groups). 3. Service interfaces (Groupable) de ne interfaces that the member objects must implement for OGS to issue callbacks to them. Group accessors and administrators are service objects. Performing a multicast to the group initiates a distributed protocol between group accessors and administrators, which ensures that messages are delivered to the members according to some condition (e.g., total order). The Groupable interface must be implemented by application objects. The IDL interfaces of the group services are described in [Fel98]. OGS interfaces are de ned in two modules: the rst one, mGroupAccess, is used by group clients and de nes operations to communicate with object groups; the second module, mGroupAdmin, is used by group members and de nes operations for administrating object groups. Interfaces from the mGroupAdmin module inherit from interfaces de ned in mGroupAccess. Inheritance is used because the client depends on less functionality than the full interface o ers. Figure 7 illustrates the interfaces and operations of the complete OGS speci cation in a class diagram. Inheritance relationships clearly show that server-side interfaces inherit from client-side interfaces. Invocable deliver() GroupAccessor multicast() cast() Groupable view_change() GroupAdministrator join_group() get_state() set_state() leave_group() destroy() get_view() destroy() GroupAccFactory create() GroupAdmFactory create() Creates Has references to Inherits from ApplicationInterface ... C l i e n t V i e w o f O b j e c t G r o u p s Fig. 7. Class diagram of OGS interfaces The interaction diagram of Fig. 8 illustrates the view change and state transfer mechanisms on two members. Initially, the group is empty. A rst member object joins the group (1), and receives the new view (2). A second member object joins the group (3), which is not empty anymore, leading to both a state transfer protocol (4, 5), and a view change noti cation (6). 5 OGS Con guration Invocations to object groups are performed by OGS. Client messages are sent via group accessor objects, and server messages are delivered via group administrator objects. Group accessors and administrators are service objects that form the visible part of the OGS runtime system, which is presented in this section. The application developer can con gure this runtime system in a number of ways, leading to di erent degrees of exibility, e ciency, transparency, or reliability. Examples of OGS programs using various con gurations can be found in [Fel98]. Group Admin. A Groupable Object A 1 join_group() view_change() 2 State transfer protocol Group Admin. B Groupable Object B 3 join_group() view_change() 6 Host A Host B get_state() 4 view_change() 6 set_state() 5 Fig. 8. Interaction diagram of view change and state transfer To conceal e ciency and exibility, OGS provides two execution models: a linkable model and a daemon model. In the rst model, the service objects are co-located with application objects, i.e., they are linked with the application and they execute in the same address space (or process). In the second model, the service objects are located in another process | the OGSd daemon program | which may be on the local or on a remote host. The linkable version of OGS is provided as a C++ dynamic library (OGSl) to be linked with C++ applications, or as a set of Java classes usable from Java applications (see Fig. 9). This execution model is more e cient since inter-process communications are more costly than invocations between objects located in the same process [GFGM98]. However, it enforces the code of the application to be written with the same programming language as the library and to support multi-threading. Client OGS Library ORB Library Server OGS Library ORB Library Multicast Message Fig. 9. The co-located execution model The daemon execution model, with two separate processes, has the advantage of decoupling the service from the application, enabling several applications running on the same host to use the same resources. It also allows user applications written in another programming language, such as Smalltalk, to use the C++ or Java service. Figure 10 illustrates the use of the OGS daemon on the client side, while the service is linked with the application on the server side.6 OGS DAemon ORB Library Remote Invocation Client ORB Library Server OGS Library ORB Library Multicast Message Fig. 10. The remote execution model The co-located execution model mainly bene ts from its e ciency, while the remote execution model provides language heterogeneity. But the choice of the execution model also a ects reliability. As matter of fact, with the co-located execution model, the application does not need to care about potential link failures or crashes of service objects, since OGS cannot fail independently from the application. If the application and the service objects are located on di erent machines, the application must handle network, machine, and OGS daemon process failures. Therefore, using a remote daemon requires the communication link between the application and the daemon to be reliable (this hypothesis is realistic in a local area network). 6 OGS Performances This section presents the performance of the OGS implementation on VisiBroker [Vis98] (some performance measurements of OGS on Orbix [ION97] may be found in [FGS98]).7 We focus on client multicast invocations, i.e., the cost of invocations through OGS, from a client to a group of objects. We compute the overhead of using OGS over using plain CORBA invocations, and we investigate the source of this overhead. These measurements focus on OGS performance. They do not analyze the intrinsic cost of invocations going through the ORB in detail. Exhaustive performance measurements and analysis of CORBA latency with di erent ORBs may be found in [GS98]. 6 Note that the servers could also use an OGS daemon. 7 Due to some limitations in the current version of Orbix, we could not perform all the tests presented in this section with the Orbix version of OGS. 6.1 System Con guration Our performance measurements have been performed with the C++ version of OGS, compiled with VisiBroker 3.2. Testing took place on a local 10 Mbit Ethernet network, interconnecting 13 Sun SPARCstations running Solaris 2.5.1 or 2.6, under normal load conditions (all workstations were running X Windows, as well as several user applications such as netscape or emacs). Among these workstations, there were four Sun UltraSPARC 30 (250 Mhz processor, 128 MB of RAM), and nine Sun UltraSPARC 1 (170 Mhz processor, 64 MB of RAM). For tests involving up to four hosts, only the UltraSPARC 30 workstations were used. All the client and server applications were located on di erent hosts, except the OGS daemon which was located on the same host as the client. The tests have been run with the TCP-no-delay option that sets all sockets to immediately send requests, instead of bu ering them and sending them in batches. 6.2 Test Scenarios Our performance tests evaluate the latency of multicast invocations issued by a client to an object group when no failure occurs. These invocations use the various semantics provided by OGS: total order (using two di erent algorithms [Fel98]), reliable, and unreliable; and three di erent modes of invocations: untyped invocations with the OGS library, untyped invocations with the OGS daemon, and typed invocations with the OGS daemon. The group size varies from one to ten members. The client waits for a single reply from the servers, except with the optimistic active replication algorithm, with which the client waits for a majority of replies. The test program operates as follows: a single client executes several rounds, in each of which it issues a xed number of synchronous invocations (typically 100). The client waits for a reply from each request before issuing the next invocation. The total time of each round is divided by the number of invocations issued during the round to obtain the latency of a single invocation. We kept the value of the best round. Since there is only one client, invocations are not performed concurrently and OGS cannot bene t from its consensus-based total order algorithm that can order several requests at once. Therefore, this test is not a good measure of the total throughput of OGS with this algorithm. 6.3 Performances Measurements In Sect. 4.2, we have classi ed transparency according to three categories: behavior transparency, plurality transparency, and type transparency. We analyze the costs of the OGS architecture according to this classi cation: we rst evaluate the cost of the various multicast primitives of OGS (behavior); plurality transparency is analyzed in details in [Fel98]; we nally evaluate the performance overhead induced by the use of dynamic typing facilities in OGS (type). The Cost of Behavior. The semantics of multicast invocations depend on the behavior of an object group. Figure 11 illustrates the cost of the di erent OGS untyped invocation primitives, with the library execution style and di erent group sizes. 0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 1 2 3 4 5 6 7 8 9 10 Number of Members La te nc y (m s. /in v. ) Total Order Opt. Active Repl. Reliable Unreliable Fig. 11. Comparing OGS multicast primitives This gure shows that the total order and reliable multicast primitives grow faster than the other primitives. This is due to the fact that the former primitives are based on a simple reliable multicast algorithm; the complexity of the number of messages for this algorithm is O(n2) for n participants.8 In contrast, the optimistic active replication algorithm [Fel98] has been optimized so that is does not use the same reliable multicast primitive when there is no failure. Its cost grows thus linearly, similarly to the unreliable multicast primitive. Note that the cost of an invocation to a single object issued directly through VisiBroker is about three times lower than the same invocation going through OGS to a group with a single member. The Cost of Typing. Type transparency is an important feature of OGS because it hides groups from the application developer, and makes it possible to reuse existing applications without having to modify the client. In the current version of OGS, typed communication is available only for the daemon execution 8 Note that this cost could be reduced by using a \smarter" algorithm. style. Figure 12 compares the latency of untyped totally ordered requests (library and daemon execution styles) with that of typed requests. 0.00 20.00 40.00 60.00 80.00 100.00 120.00 1 2 3 4 5 6 7 8 9 10 Number of Members La te nc y (m s. /in v. ) Untyped Library Untyped Daemon Typed Daemon Fig. 12. Untyped vs. typed communicationThis gure illustrates that there is a xed overhead of about 1 millisecond forusing the daemon. This corresponds to the latency of a single two-way invocationthrough the ORB. The typed version of OGS adds an overhead of about 20milliseconds. This overhead results from the use of the DSI and the DII for typetransparency, and is independent of the group size. This is due to the fact thatthe DSI and the DII are used only once on the client and the server side.Another source of overhead comes from request management. When pro lingOGS, we noticed that a non-negligible part of the time required for remoteinvocations is spent in constructing requests. We also observed that workingwith untyped any values has a signi cant impact on performance. Unlike otherIDL types, any values are augmented by a typecode information that containsdetails about the actual type of the value. This information increases the size ofthe messages sent on the network. Moreover, validity checks upon data extractionslow down the remote invocation process.7 ConclusionDistributed computing is one of the major trends in the computer industry. Assystems become more distributed, they also become more complex and have to deal with new kinds of problems, such as partial crashes and link failures. Manyof these problems can be solved by replicating critical components.To answer the growing demand in distributed technologies, several middle-ware environments have emerged during the last few years. These environmentshowever lack support for one-to-many communication primitives, which are nec-essary to implement object replication. Augmenting a middleware architectureby adding support for object groups provides straightforward support for repli-cating distributed objects.We have presented here an open architecture for object group support inCORBA, based on a component-oriented approach. This architecture is genericand can be applied to other middleware environments than CORBA. The Ob-ject Group Service (OGS) de nes an object group model and speci es a set ofinterfaces adapted to group communication in middleware environments. OGSis decomposed into several components that conspire to provide higher-level ser-vices. This decomposition promotes modularity and reusability, and complieswith the CORBA architectural model.OGS provides support for dynamic group membership and for group mul-ticast with various reliability and ordering guarantees. In addition, OGS pro-poses several execution styles and various levels of transparency. CORBA ob-jects can easily be made groupable using interface inheritance. Client applica-tions can communicate with object groups in the same way as they do withsingleton objects. A prototype implementation of OGS is freely available athttp://lsewww.epfl.ch/OGS/.References[Bir93]K.P. Birman. The process group approach to reliable distributed com-puting. Communications of the ACM, 36(12):36{53, December 1993.[CHY+98] P.E. Chung, Y. Huang, S. Yajnik, D. Liang, and J. Shih. Doors: Providingfault tolerance for corba applications. In IFIP International Conferenceon Distributed Systems Platforms and Open Distributed Processing (Mid-dleware'98), September 1998.[CT96]T.D. Chandra and S. Toueg. Unreliable failure detectors for reliabledistributed systems. Journal of the ACM, 43(2):225{267, 1996.[CZ85]D.R. Cheriton and W. Zwaenepoel. Distributed process groups in theV kernel. ACM Transactions on Computer Systems, 3(2):77{107, May1985.[Fel98]P. Felber. The CORBA Object Group Service: A Service Approach toObject Groups in CORBA. PhD thesis, Ecole Polytechnique Federale deLausanne, Switzerland, 1998. Number 1867.[FGG96] P. Felber, B. Garbinato, and R. Guerraoui. The design of a CORBA groupcommunication service. In Proceedings of the 15th IEEE Symposium onReliable Distributed Systems, pages 150{159, October 1996.[FGG97] P. Felber, B. Garbinato, and R. Guerraoui. Special Issues in Object-Oriented Programming, chapter Towards Reliable CORBA: Integrationvs. Service Approach, pages 199{205. dpunkt-Verlag, 1997. [FGS98]P. Felber, R. Guerraoui, and A. Schiper. The implementation of aCORBA object group service. Theory and Practice of Object Systems,4(2):93{105, 1998.[GFGM98] R. Guerraoui, P. Felber, B. Garbinato, and K. Mazouni. System supportfor object groups. In Proceedings of the ACM Conference on Object Ori-ented Programming Systems, Languages and Applications (OOPSLA'98),1998.[GS98]A. Gokhale and D. Schmidt. Measuring and optimizing CORBA latencyand scalability over high-speed networks. IEEE Transactions on Com-puters, 47(4):391{413, April 1998.[HT93]V. Hadzilacos and S. Toueg. Distributed Systems, chapter 5: Fault-Tolerant Broadcasts and Related Problems, pages 97{145. Addison-Wesley, 2nd edition, 1993.[II94]IONA and Isis. An Introduction to Orbix+Isis. IONA Technologies Ltd.and Isis Distributed Systems, Inc., 1994.[ION97]IONA. Orbix 2.2 Programming Guide. IONA Technologies Ltd., Mar1997.[Maf95]S. Ma eis. Run-Time Support for Object-Oriented Distributed Program-ming. PhD thesis, University of Zurich, February 1995.[MMSA+96] L.E. Moser, P.M. Melliar-Smith, D.A. Agarwal, R.K. Budhia, and C.A.Lingley-Papadopoulos. Totem: A fault-tolerant multicast group commu-nication system. Communications of the ACM, 39(4):54{63, April 1996.[MMSN98] L.E. Moser, P.M. Melliar-Smith, and P. Narasimhan. Consistent objectreplication in the Eternal system. Theory and Practice of Object Systems,4(2):81{92, 1998.[OMG97] OMG. CORBAservices: Common Object Services Speci cation. OMG,1997.[OMG98a] OMG. The Common Object Request Broker: Architecture and Speci ca-tion. OMG, February 1998.[OMG98b] OMG. Fault tolerant CORBA Using Entity Redundancy, Request ForProposal. OMG, Apr 1998. http://www.omg.org/techprocess/meetings/schedule/Fault Tolerance RFP.html.[Ses97]R. Sessions. COM and DCOM: Microsoft's Vision for Distributed Objects.John Wiley & Sons, 1997.[Vis98]Visigenic. VisiBroker for C++ 3.2 Programmer's Guide. Visigenic Soft-ware, Inc., Mar 1998.[vN56]J. von Neumann. Probabilistic logics and the synthesis of reliable organ-isms from unreliable components. In C. E. Shannon and J. McMarthy,editors, Automata Studies, pages 43{98. Princeton University Press, 1956.